Skip to content
This repository has been archived by the owner on Feb 15, 2024. It is now read-only.

Token should not be able to have whitespace #24

Open
schmmd opened this issue Oct 11, 2013 · 1 comment
Open

Token should not be able to have whitespace #24

schmmd opened this issue Oct 11, 2013 · 1 comment

Comments

@schmmd
Copy link
Member

schmmd commented Oct 11, 2013

Unfortunately, BreezeSentencer uses Tokenizer.computeOffsets to compute offsets from the resulting sentences, so simply adding require(string.forall(!_.isWhitespace)) breaks BreezeSentencer.

@schmmd
Copy link
Member Author

schmmd commented Oct 11, 2013

@rbart, @jgilme1 postag and chunk no longer can contain whitespace FYI. b4fdb99

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant