We’ve completed the dcc-release pipeline, at least as far as export.
At the same time, we have deployed an instance of dcc-download-server and have it communicating to the HDFS file system. We’ve manually populated a directory with the contents of
dcc-download-server/src/test/resources/fixtures/input. We’ve configured proxies and can now download data from HDFS successfully.
As far as I can tell the dcc-release process is incomplete, in that it does not create a directory structure within HDFS that dcc-download expects.
There is code here in dcc-etl that looks like it might create the expected directory.
Is there any guidance you can share on how to prepare data for handoff between dcc-release and dcc-download?
Thanks very much for reading.