Documentation
¶
Index ¶
- Constants
- func AnalyzeTextContent(data []byte) (string, error)
- func DetectByContent(data []byte) (string, error)
- func DetectByExtension(filename string) string
- func DetectFileType(data []byte, filename string) (string, error)
- func LooksLikeCSV(data []byte) bool
- func ProcessCSV(ctx context.Context, uploadID, source string, reader io.Reader, ...) (int, error)
- func ProcessJSON(ctx context.Context, uploadID, source string, reader io.Reader, ...) (int, error)
- func UploadExternalData(ctx context.Context, source string, reader io.Reader, filename string, ...) (string, int, error)
- type StoreFunc
Constants ¶
View Source
const ( DefaultBatchSize = 500 DefaultWorkerCount = 4 DefaultBufferSize = 64 * 1024 // 64KB buffer DefaultChannelSize = 1000 ContextCheckInterval = 100 )
Configuration constants for performance tuning
Variables ¶
This section is empty.
Functions ¶
func AnalyzeTextContent ¶
func DetectByContent ¶
func DetectByExtension ¶
func LooksLikeCSV ¶
func ProcessCSV ¶
func ProcessCSV(ctx context.Context, uploadID, source string, reader io.Reader, store StoreFunc) (int, error)
ProcessCSV now uses parallel processing with workers
func ProcessJSON ¶
func ProcessJSON(ctx context.Context, uploadID, source string, reader io.Reader, store StoreFunc) (int, error)
ProcessJSON now uses batch processing
func UploadExternalData ¶
func UploadExternalData(ctx context.Context, source string, reader io.Reader, filename string, store StoreFunc) (string, int, error)
UploadExternalData handles the process of uploading external data by detecting file type, parsing, and storing it. Parameters: - ctx: The context for controlling execution. - source: The source of the external data. - reader: An io.Reader for reading the data. - filename: The name of the file being uploaded. - store: The callback function to store processed transactions. Returns: - string: The ID of the upload. - int: The total number of records processed. - error: If any step of the process fails.
Types ¶
Source Files
Click to show internal directories.
Click to hide internal directories.