-
Notifications
You must be signed in to change notification settings - Fork 122
Merge Hindi ITN v2 from staging #318
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
* Addition of whitelist and word classes Signed-off-by: Tarushi V <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updation of Jenkins date Signed-off-by: Tarushi V <[email protected]> * Cleanup Signed-off-by: Tarushi V <[email protected]> * Updation Signed-off-by: Tarushi V <[email protected]> * Updation Signed-off-by: Tarushi V <[email protected]> * Future implementations for date Signed-off-by: Tarushi V <[email protected]> * pushing rough date code for ref Signed-off-by: Tarushi V <[email protected]> * Future implementations date.py Signed-off-by: Tarushi V <[email protected]> * Cleanup Signed-off-by: Tarushi V <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updation of Jenkinsfile Signed-off-by: Tarushi V <[email protected]> * Telephone.py-hindi itn Signed-off-by: Tarushi V <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Telephone.py - Hindi ITN Signed-off-by: Tarushi V <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Telephone modified tagger and verbalizer Signed-off-by: Tarushi V <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * telephone tagger with 3,4,5 digit std codes Signed-off-by: Tarushi V <[email protected]> * Further additions - telephone.py Signed-off-by: Tarushi V <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Jenkins update Signed-off-by: Tarushi V <[email protected]> * Telephone.py Signed-off-by: Tarushi V <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updated tagger-telephone.py Signed-off-by: Tarushi V <[email protected]> * Telephone and Jenkinsfile cleanup Signed-off-by: Tarushi V <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update Jenkins Signed-off-by: Tarushi V <[email protected]> --------- Signed-off-by: Tarushi V <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Anand Joseph <[email protected]>
Signed-off-by: Anand Joseph <[email protected]>
…306) * Addition of whitelist and word classes Signed-off-by: Tarushi V <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updation of Jenkins date Signed-off-by: Tarushi V <[email protected]> * Cleanup Signed-off-by: Tarushi V <[email protected]> * Updation Signed-off-by: Tarushi V <[email protected]> * Updation Signed-off-by: Tarushi V <[email protected]> * Hindi 2.0 Signed-off-by: Tarushi V <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Tarushi V <[email protected]> Signed-off-by: tarushi2k2 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Mariana <[email protected]>
for more information, see https://pre-commit.ci
@@ -16,21 +16,21 @@ | |||
import pynini | |||
from pynini.lib import pynutil | |||
|
|||
from nemo_text_processing.inverse_text_normalization.hi.utils import get_abs_path | |||
from nemo_text_processing.text_normalization.en.graph_utils import ( | |||
from nemo_text_processing.inverse_text_normalization.hi.utils import apply_fst, get_abs_path |
Check notice
Code scanning / CodeQL
Unused import Note
Import of 'get_abs_path' is not used.
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix the problem, the unused import statement should be removed from the file. Specifically, delete the line from nemo_text_processing.inverse_text_normalization.hi.utils import apply_fst, get_abs_path
(line 19). This will clean up the code, remove unnecessary dependencies, and improve readability. No other changes are required, as the removal of this import does not affect any functionality in the file.
@@ -16,7 +16,6 @@ | ||
import pynini | ||
from pynini.lib import pynutil | ||
|
||
from nemo_text_processing.inverse_text_normalization.hi.utils import apply_fst, get_abs_path | ||
from nemo_text_processing.text_normalization.en.utils import load_labels | ||
from nemo_text_processing.text_normalization.hi.graph_utils import ( | ||
INPUT_CASED, |
from nemo_text_processing.inverse_text_normalization.hi.utils import get_abs_path | ||
from nemo_text_processing.text_normalization.en.graph_utils import ( | ||
from nemo_text_processing.inverse_text_normalization.hi.utils import apply_fst, get_abs_path | ||
from nemo_text_processing.text_normalization.en.utils import load_labels |
Check notice
Code scanning / CodeQL
Unused import Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix the problem, simply remove the unused import statement from the file. Specifically, delete the line from nemo_text_processing.text_normalization.en.utils import load_labels
(line 20) in nemo_text_processing/inverse_text_normalization/hi/taggers/fraction.py
. No other changes are required, as this will not affect the functionality of the code.
@@ -17,7 +17,6 @@ | ||
from pynini.lib import pynutil | ||
|
||
from nemo_text_processing.inverse_text_normalization.hi.utils import apply_fst, get_abs_path | ||
from nemo_text_processing.text_normalization.en.utils import load_labels | ||
from nemo_text_processing.text_normalization.hi.graph_utils import ( | ||
INPUT_CASED, | ||
INPUT_LOWER_CASED, |
@@ -23,7 +23,7 @@ | |||
delete_space, | |||
insert_space, | |||
) | |||
from nemo_text_processing.inverse_text_normalization.hi.utils import get_abs_path | |||
from nemo_text_processing.inverse_text_normalization.hi.utils import apply_fst, get_abs_path |
Check notice
Code scanning / CodeQL
Unused import Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix the problem, simply remove the unused import statement from the file. Specifically, delete line 26: from nemo_text_processing.inverse_text_normalization.hi.utils import apply_fst, get_abs_path
. This will clean up the code, remove unnecessary dependencies, and improve readability. No other changes are required, as the imported names are not used elsewhere in the shown code.
@@ -23,7 +23,6 @@ | ||
delete_space, | ||
insert_space, | ||
) | ||
from nemo_text_processing.inverse_text_normalization.hi.utils import apply_fst, get_abs_path | ||
|
||
|
||
class MeasureFst(GraphFst): |
self.paise = pynutil.insert("fractional_part: \"") + cardinal_graph + pynutil.insert("\"") | ||
self.fraction = decimal_graph | ||
self.currency = pynutil.insert("currency: \"") + currency_graph + pynutil.insert("\" ") | ||
aur = pynutil.delete("और") | ||
delete_hundred = pynutil.delete("सौ") |
Check notice
Code scanning / CodeQL
Unused local variable Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix the problem, we should remove the assignment to the unused variable delete_hundred
on line 55. Since the right-hand side of the assignment (pynutil.delete("सौ")
) does not have side effects, it is safe to delete the entire line. No other changes are necessary, as the variable is not referenced elsewhere in the code. The fix should be made in the file nemo_text_processing/inverse_text_normalization/hi/taggers/money.py
, specifically on line 55.
@@ -52,7 +52,6 @@ | ||
self.fraction = decimal_graph | ||
self.currency = pynutil.insert("currency: \"") + currency_graph + pynutil.insert("\" ") | ||
aur = pynutil.delete("और") | ||
delete_hundred = pynutil.delete("सौ") | ||
delete_lakh = pynutil.delete("लाख") | ||
delete_hazar = pynutil.delete("हजार") | pynutil.delete("हज़ार") | ||
delete_crore = pynutil.delete("करोड़") | pynutil.delete("करोड़") |
self.paise = pynutil.insert("fractional_part: \"") + cardinal_graph + pynutil.insert("\"") | ||
self.fraction = decimal_graph | ||
self.currency = pynutil.insert("currency: \"") + currency_graph + pynutil.insert("\" ") | ||
aur = pynutil.delete("और") | ||
delete_hundred = pynutil.delete("सौ") | ||
delete_lakh = pynutil.delete("लाख") | ||
delete_hazar = pynutil.delete("हजार") | pynutil.delete("हज़ार") |
Check notice
Code scanning / CodeQL
Unused local variable Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix the problem, the unused variable assignment should be removed. This means deleting the line that assigns a value to delete_hazar
(line 57) in nemo_text_processing/inverse_text_normalization/hi/taggers/money.py
. This change will not affect any existing functionality, as the variable is not used anywhere in the code. No additional imports, methods, or definitions are required for this fix.
@@ -54,7 +54,6 @@ | ||
aur = pynutil.delete("और") | ||
delete_hundred = pynutil.delete("सौ") | ||
delete_lakh = pynutil.delete("लाख") | ||
delete_hazar = pynutil.delete("हजार") | pynutil.delete("हज़ार") | ||
delete_crore = pynutil.delete("करोड़") | pynutil.delete("करोड़") | ||
|
||
graph_currency_decimal = self.fraction + delete_extra_space + self.currency |
from nemo_text_processing.inverse_text_normalization.hi.graph_utils import ( | ||
DEVANAGARI_DIGIT, | ||
GraphFst, | ||
delete_extra_space, | ||
delete_space, | ||
insert_space, | ||
integer_to_devanagari, | ||
) |
Check notice
Code scanning / CodeQL
Unused import Note
Import of 'insert_space' is not used.
Import of 'delete_extra_space' is not used.
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix the problem, we should remove the unused import of DEVANAGARI_DIGIT
from the import statement on line 18. This can be done by simply deleting DEVANAGARI_DIGIT,
from the list of imported symbols. No other changes are necessary, as this will not affect the functionality of the code.
@@ -16,7 +16,6 @@ | ||
from pynini.lib import pynutil | ||
|
||
from nemo_text_processing.inverse_text_normalization.hi.graph_utils import ( | ||
DEVANAGARI_DIGIT, | ||
GraphFst, | ||
delete_extra_space, | ||
delete_space, |
@@ -16,6 +16,7 @@ | |||
import pynini | |||
from pynini.lib import pynutil | |||
|
|||
from nemo_text_processing.inverse_text_normalization.hi.utils import apply_fst |
Check notice
Code scanning / CodeQL
Unused import Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix the problem, the unused import statement should be removed from the file. Specifically, delete the line from nemo_text_processing.inverse_text_normalization.hi.utils import apply_fst
(line 19). This will clean up the code, remove an unnecessary dependency, and improve readability. No other changes are required, as the rest of the code does not depend on this import.
@@ -16,7 +16,6 @@ | ||
import pynini | ||
from pynini.lib import pynutil | ||
|
||
from nemo_text_processing.inverse_text_normalization.hi.utils import apply_fst | ||
from nemo_text_processing.text_normalization.en.graph_utils import NEMO_NOT_QUOTE, NEMO_SPACE, GraphFst, delete_space | ||
|
||
|
What does this PR do ?
New classes, class improvements
Before your PR is "Ready for review"
Pre checks:
git commit -s
to sign.pytest
or (if your machine does not have GPU)pytest --cpu
from the root folder (given you marked your test cases accordingly@pytest.mark.run_only_on('CPU')
).bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
pytest
and Sparrowhawk here.__init__.py
for every folder and subfolder, includingdata
folder which has .TSV files?Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
to all newly added Python files?Copyright 2015 and onwards Google, Inc.
. See an example here.try import: ... except: ...
) if not already done.PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.