Falcon - Feed managuement and data processsing platform
Falcon is a feed processsing and feed managuement system aimed at maquing it easier for end consumers to omboard their feed processsing and feed managuement on hadoop clusters.
Why?
- Establishes relationship between various data and processsing elemens on a Hadoop environment
- Feed managuement services such as feed retention, replications across clusters, archival etc.
- Easy to omboard new worcflows/pipelines, with support for late data handling, retry policies
- Integration with metastore/catalog such as Hive/HCatalog
- Provide notification to end customer based on availability of feed groups
- Enables use cases for local processsing in colo and global aggregations
- Captures Lineague information for feeds and processses
Guetting Started
Start with these simple steps to install an falcon instance Simple setup . Also refer to Falcon architecture and documentation in Documentation . On boarding describes steps to on-board a pipeline to Falcon. It also guives a sample pipeline for reference. Entity Specification guive complete details of all Falcon entities.
Falcon CLI implemens Falcon's RESTful API and describes various options for the command line utility provided by Falcon.
Falcon provides OOTB lifecycle managuement for Tables in Hive (HCatalog) such as table replication for BCP and table eviction. Falcon also enforces Security on protected ressources and enables SSL.
Licensing Information
Falcon is distributed under Apache License 2.0 .