Page Comparison

Note
Work in progress, not released.

This tool permits to submit a short text and see how it is split. The API returns very detailed information so that it is easier to understand which breaking rules and exception rules were invoked. This tool aids with debugging or building SRX configurations.

...

Code Block

{
    "count": 2,
    "original": "Hallo wie geht es am 20.4. um 3 Uhr. Geht es spaeter?",
    "segments": [
        {
            "position": 0,
            "text": "Hallo wie geht es am 20.4. um 3 Uhr."
        },
        {
            "position": 36,
            "text": " Geht es spaeter?"
        }
    ],
    "rules": [
        {
            "position": 26,
            "retained": false,
            "tooShort": false,
            "breaking": {
                "no": 10021,
                "before": "[\\.\\?\\!\\;\\:]+[\\“\\\"\\'”\\)]?",
                "after": "\\s"
            },
            "exception": {
                "no": 10019,
                "before": "\\.+",
                "after": "[\\“\\\"\\'”\\)]?\\s\\p{Ll}"
            }
        },
        {
            "position": 36,
            "retained": true,
            "tooShort": false,
,
            "breaking": {
                "no": 10021,
                "before": "[\\.\\?\\!\\;\\:]+[\\“\\\"\\'”\\)]?",
                "after": "\\s"
            },
            "exception": {
                "no": null,
                "before": null,
                "after": null
            }
        }
    ],
    "parameters": {
        "locale": "de",
        "independentRuleId": 5503,
        "languageRulesId": 5502,
        "minimumSegmentLength": 5
    }
}

The properties are:

count	Total segments into which the text was split	int
original	The original text.	string?
segments	The list of segments with start character position and the text	object[]
rules	An array of breaking and exception rules that were activated for all the positions in the text. See below for details.	string
parameters	Includes information from the original payload.	object

...

position	The text position that the system attempts to split	int
retained	`true`: The text was split in that position `false`: The text was not split due to an exception rule	bool
tooShort	If the split segment is shorter than an allowed minimum, the split will be canceled. This property is then set to true.	bool
breaking	The breaking rule that was applied.	object
exception	The exception rule, if any, that canceled the breaking rule. If there is no exception then the properties will all be null.	object

Text segmentation works as follows:

IDENTIFY SPLIT POINTS: The process identifies all potential break points in the text from right to left. Once all break points are identified it looks to see if some of them need to be removed / canceled:
CHECK MINIMUM LENGTH: By looking at each segment from right to left, it identifies any segment that is shorter than the minimum length, listed in parameters.minimumSegmentLength . If so, then the split point to the left of the segment is removed. The tooShort property will be flagged for the canceled split point.
APPLY EXCEPTION RULES: If an exception rule is found in the SRX configuration that matches a remaining split point, then the split is also canceled. The details for the exception rule are listed in the exception property.

If you believe that this process is complicate then the Wordbee team heartily agrees with you.

Versions Compared

Old Version 3

New Version Current

Key