You can use the AWS Glue catalog source to extract metadata from an AWS Glue source system.
AWS Glue is a serverless ETL (extract, transform, and load) service that helps discover, prepare, and integrate data from multiple sources for analysis, machine learning, and application development.
You use Amazon Athena to query databases and tables created in AWS Glue. You can also use Amazon Athena to create schemas to use in AWS Glue.
Extracted metadata
You can extract specific metadata from AWS Glue source system with AWS Glue catalog source.
Metadata Command Center extracts the following metadata from the AWS Glue source system:
•Calculation
•Job
•Job instance
Compatible functionalities
AWS Glue offers integration with a diverse range of modules and the Python programming language.
You can use AWS Glue with the following Python functionalities:
•Standard language constructions
•Standard built-in functions
•Partially-compatible modules:
Note: Data Governance and Catalog
processes only a subset of library functions of partially-compatible modules.
- abs
- adal
- argparse
- array
- ast
- awsglue
- azure
- base64
- binascii
- calendar
- codecs
- collections
- concurrent
- contextlib
- contextvars
- copy
- copyreg
- csv
- dataclasses
- datetime
- decimal
- delta
- difflib
- distutils
- email
- enum
- errno
- fnmatch
- fractions
- functools
- gc
- genericpath
- gettext
- glob
- graphframes
- hashlib
- heapq
- hmac
- importlib
- inspect
- io
- itertools
- json
- keyword
- locale
- logging
- math
- matplotlib
- nt
- numbers
- numpy
- operator
- os
- pandas
- pathlib
- pickle
- pkgutil
- posix
- posixpath
- pprint
- py4j
- pyodbc
- pyspark
- pytz
- random
- re
- reprlib
- requests
- seaborn
- secrets
- shutil
- simplejson
- six
- sklearn
- smtplib
- socket
- ssl
- stat
- string
- struct
- subprocess
- sys
- teradatasql
- textwrap
- threading
- time
- traceback
- types
- typing
- urllib
- urllib3
- uuid
- warnings
- weakref
- xml
- yaml
- zipfile
- zlib
•Custom libraries
Note:
Custom libraries are libraries created by a user. You can also use a WHL file for your custom library.
If the catalog source detects an incompatible function or library, it can't process the statement. It skips the statement and continues to process the next one.