Meta Releases Latest Version of Code Llama 70B on Jan 2024

Meta has unveiled the latest version of Code Llama 70B build on Llama 2 family on January 29, 2024. Code Llama is a specialized version of Llama 2 and has been trained on code specific dataset of Llama 2. Meta in its attempt to foster AI development has built Code Llama specifically for code generation supporting most popular languages like Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash with four different versions, catering to different requirements.  

Code Llama 70B is touted as the best performing and most capable Llama family model for code generation.

Key features:

1. Different Versions free for commercial and research use:

Code Llama has been released with the same versions as already released Code Llama models:

• Code Llama – 70B, the foundational code model;
• Code Llama – 70B – Python, 70B specialized for Python;
• Code Llama – 70B – Instruct 70B, which is fine-tuned for understanding natural language instructions.

Open-Source Availability

Since its release, Meta made it a point to make all the versions of Llama LLM free to use for commercial as well as research purposes. Meta believes in an open ecosystem to foster development and adoption of technology. As we discussed in our previous blog, one of the best search engines with AI integration perplexity.ai has been developed based on Llama open-source model.

• Different Size Models as per the latency and serving requirements:

Meta has released four different sized models of Code Llama with 7B, 13B, 34B and 70B. Models with 7B, 13B and 34B have been trained on 500 Billion tokens of code and code related data whereas 70B sized model has been trained on 1 Trillion tokens. The 7B, 13B and instruct have been trained with fill in the middle (FIM) capability which can be used to insert code into existing code. Each model caters to different serving and latency requirements.

A single GPU is sufficient to serve 7B model where latency requirement is low and faster for applications like real-time code completion. For optimal results and coding assistance 34B and 70B models are best suited.

• Enhanced Performance:

As compared to other available LLMs, the recent release of Code Llama performs better in code generation as shown below for two important coding benchmarks HumanEval and Most Basic Python Programming (MBPP). HumanEval tests the model’s ability to complete code based on docstrings and MBPP tests the model’s ability to write code based on a description.

Code Llama 70B -Instruct outperforms other LLMs in HumanEval and Multilingual HumanEval coding benchmarks and Code Llama-Python 70B has achieved score of 65.6 on MBPP coding benchmark.

• Higher Safety:

Advancements in Artificial Intelligence are bound to generate greater risks of being misused for exploiting the existing systems. Meta made sure to follow safety guidelines with red teaming efforts by running a quantitative evaluation of Code Llama’s risk of generating malicious code.

All the details about red teaming efforts about malware development, offensive security engineering, responsible AI and software engineering are available in the research paper.

Code Llama for Programmers:

The goal to release enhanced version is to assist programmers in assisting various tasks like writing new code, developing new application and debugging existing code.

1. Better Productivity:

Programmers will have more time to focus on the human centric aspect of their job making it more efficient by reducing the burden of repetitive tasks. Code Llama fast tracks workflow development, giving more time for creative problem solving. Programmers are already using LLMs for myriad of programming tasks and enhanced version of Code Llama will surely act as a booster for better productivity.

1. Fostering Possibilities:

Code Llama can also analyze existing code, detect bugs and suggest improvements to the existing code on top of code generation. It will ultimately result in creating more robust and efficient software.  This will also help in better utilization of software testing to look for potential loopholes in generated codes.

1. Lowering Entry Barrier in Coding:

For beginners, coding seems to be a daunting task but with assistance from Llama even beginners can complete daunting tasks. At times this little assistance in the beginning can greatly enhance the learning curve not only for beginners but also for professionals. Overall entry barriers for programmers are reduced with enhanced code writing and generation with debugging abilities.

To further foster the collaboration between AI and Llama 2, developers can focus on few key areas:

Contextual Awareness: Giving Llama the ability to comprehend the particular context of a coding activity, including the project specifications and the current codebase will help in contextual awareness.

Domain Specialization: Improving accuracy and relevance by fine-tuning the model for particular programming languages and domains.

The possibilities of utilizing enhanced Code Llama are endless and completely depend upon the end user application and potential solutions which are customizable with open-source architecture of Llama models. Moreover, open-source models further foster code development with the right checks and balances for safety.

For more details you can access: https://ai.meta.com/blog/code-llama-large-language-model-coding/?utm_source=llama_updates&utm_medium=email&utm_campaign=codellama

Code Llama’s training recipes are available on: Github repository.

To Download Llama 2 Model weights