Module textindex

Expand description

Text extraction for the full-text search over file contents.

STEP (.step, .stp, .p21) and IGES (.iges, .igs): the quoted strings of the file – product names, descriptions, authors, organisation, layer and entity names – without the numeric geometry.
Plain text formats (txt, md, csv, json, xml, yaml, log, Teamcenter PLM XML, OpenSCAD, motor thrust curves, …): the text itself, line by line, with its decimal numbers – a text or CSV report reads like a PDF one (charset: byte order mark, (mostly valid) UTF-8, else Windows-1252).
Geometry as text (ASCII STL, OBJ, KiCad boards and symbols): the words without the decimal numbers (coordinates are noise in a search).
PDF: the text layer, read in an isolated process (see pdftext); scanned PDFs have none.
HTML (.html, .htm, the reports of FEA/CFD tools): the readable text with table rows as lines, read in the same isolated process (see htmltext).
Everything else (binary CAD, images): nothing (not indexed yet).

At most MAX_READ bytes are read and MAX_TEXT characters are kept.

Constants§

TAIL_MARK: Between the beginning and the end of a file whose middle was left out.

extract: Extracts searchable text from raw file content.
extract_file: Reads (the beginning of) a stored file and extracts its text. A large plain-text file (solver log, print output) keeps its end as well: the text of the beginning, the marker TAIL_MARK and the text of the last TAIL_READ (64 KiB) bytes – where a solver writes its status and final values. The end is only added when the beginning was text and the end is text in the same encoding (no NUL, no UTF-16 byte order mark at the start).
indexes: Whether the content of such a file is indexed for the full-text search.
prefix_query: Builds a prefix tsquery (flan:* & m6:*) from free text, or None if the input contains no searchable word.
read_tail: The last max bytes of an open file, starting at a line: the first, cut line is dropped (unless the whole file fits). Never reads more than max bytes, whatever the file’s size; empty on any error. It takes the file the caller already opened (seeking it), so a file is opened once.