Start a new topic

what's the relationship between the slate toolkit and the moses

I was linked to this slate toolkit while I was learning how to install the Moses. However I can not find out the relationship between the above. So, is there somebody who could help me out. There is really little I can find on the search engines.


thx~

1 Comment

Thank you, Tammy.


Let's look at this in terms of two options, "A" and "B".


Option A - do it yourself


Moses (http://statmt.org/moses/) is a toolkit of open source tools (utilities) intended to make it easy for academic researchers to do their work. To use if for business, you must download its source code from this URL: https://github.com/moses-smt/mosesdecoder and compile executable utilities. The Moses toolkit itself, however, is not a complete and fully functional statistical machine translation (SMT) system. It's an important part of a complete SMT system, but it is not the SMT system.


To have a complete SMT system, you must also make sure that your host computer has the required GNU utilities. You must also download, compile and install the MGIZA++ project (https://github.com/moses-smt/mgiza). Additionally, there are dozens of other undocumented dependencies that, if absent, will cause your SMT system to fail. It requires considerable C++ programmer skills and Linux expertise to make sure you have Moses and all its dependencies in the correct place for the SMT system to find and use everything it needs. Without the necessary expertise, it could take you hours, days or even weeks of learning before you can create a fully-functional SMT system that is ready for you to use.


With Option A, you need average C++ programming skills to compile/build the SMT toolkit on Linux and MacOS. On MacOS, you must do additional work to identify and resolve many more undocumented dependencies. You need expert C++ skills on both Linux and Windows, before you can create a Windows version. Even then, it will likely take you several months of learning and work before you have a functioning SMT system on Windows.


Option B - packaged distribution


Slate Toolkit Edition (https://slate.rocks/slate-toolkit-edition/) is a package (zip file) with open source utilities that we compiled and distribute for free. It includes the Moses toolkit, MGIZA++, the GNU utilities and all of the undocumented dependencies that constitute a fully-functional SMT system. Essentially, we did the steps in Option A for you.


With Option B, you simply download and unzip the Windows package, the Linux package or both. When you unzip the package, everything is in the right place and ready for you to use.


Linux (& MacOS) vs Windows


But please be careful, "ready to use" might not mean what you expect it to mean. So, here's what it means in the context of the academic researchers who created Moses for their  work.


A "ready to use" system on Linux & MacOS (both option A & option B) includes hundreds of utilities and scripts that academic researchers created for various experiments and tests. Each of them are in various states of functionality or disrepair, depending on how the researcher last worked on the experiment. The Moses team tested and maintained the functionality for only a small subset of utilities and scripts. These constitute less than 30 utilities and scripts that are necessary to build a baseline SMT system and some popular variants. 


A "ready to use" DIY system on Windows (option A) will have the same hundreds of utilities and scripts. Most of them are guaranteed not to run on MS Windows. This is because the researchers exclusively work with Linux (and MacOS). When building your DIY system, you are responsible for sorting through which utilities and scrpts to see which work.


A "ready to use" Slate Toolkit Edition system on Windows (option B) has fewer utilities and scripts. We do not package many of the components that do not run on Windows. The system includes all utilities and scripts that are necessary to create and use phras-based statistical machine translation models. For more information, see the README file.


What does "ready for you to use" mean?


Whether you choose the Moses DIY approach or Slate Toolkit package, after you complete Option A or Option B processes, you will have a "toolkit." This is not the same as a finished application. If you are a programmer on MS Windows, a toolkit is similar to software developer's kit (SDK) but less... Really, because there's no documentation and no programming interface.


The toolkit is a collection of command-line tools (or utilities). This means you must open a terminal (Windows Command Prompt) and manually type commands that set 20 or 30 options for each one. These commands can be as long as 200+ characters. If you type a mistake, you get an error and must start again.


Slate Toolkit, however, also includes shell scripts (10 Bash scripts on Linux or 10 Batch files on Windows) that are unique to Slate Toolkit. These 10 scripts represent the absolute bare minimum processing stages that you must complete before you have a functional SMT model that can generate translations. Fully mature end-user SMT applications consist of dozens or hundreds of extra steps that enable various features in addition to the bare minimum 10 steps.


Summary


So, back to your question, "What's the relationship between the Slate Toolkit and Moses?" There is a LOT of overlap between the two with their core utilities and scripts, but they are not the same. Each includes utilities and scripts the other does not. They require different skill levels to setup before you can use them. Once set up, the baseline core functions are the same but diverge in what extra features are included. In that regard, you need one or the other but not both.


You might find it helpful to review the features table that compares Slate Toolkit with the other Slate products: https://slate.rocks/slate-rocks/compare-slate-products/ 


I hope this helps. You can find technical details about what's in Slate Toolkit Edition on the "Features" and "README" tabs on the product page: (https://slate.rocks/slate-toolkit-edition/).


Thank you for asking, Tammy.

Tom


Login or Signup to post a comment