This feature is currently in Walk.
DISCO has recently completed a set of improvements to Slack processing. Not only can DISCO support larger Slack exports overall, we can also process more documents per minute. For more information on our existing Slack processing, please see Chat Streams: Slack – DISCO.
Chat data, such as Slack, is different from other data types because the relationship between input files and output documents is many-to-many instead of one-to-one. For example, multiple source Slack json files may contribute to the messages and metadata of just one 24-hour Slack document within DISCO after ingestion. Likewise, one source Slack json file may be split to create multiple 24-hour Slack documents within DISCO.
To better support this unique structure and give users more insight into document processing, DISCO has updated the validation reporting for Slack ingests.
The existing validation report only surfaced errors, requiring users to guess what the impact or outcome was. The new validation reporting eliminates this guesswork by explicitly stating what the errors were, what output documents are impacted, and what input documents are impacted.
When reviewing the new validation report for Slack ingests, users will find the following enhancements:
- The validation report will provide a comprehensive summary of source files and indicate whether they were successful, failed, duplicate, partially processed, or unused. Some files in chat export formats, such as integration_logs.json in a Slack export, will be denoted as unused as they do not turn into chat documents.
- Exception reports now include specific details about which output chat documents were impacted by the warnings and/or errors.
- Utilize the ObjectId column in the validation report to connect back to source Slack events - this is the ts field for messages, and the id field for other objects.
- In the event that files fail and cannot be processed as chat data, DISCO will automatically process those failed files as native files.
- Impacted DISCO documents can be investigated in detail through the ingest or error report