From Prompts to Code: A Side-by-Side Comparison of AI Code Generation Engines
Technical Blog
AI is the new hot topic in the computing world. As a developer, I use AI daily to help with my coding. It’s a wonderful tool for quickly getting answers or sometimes coding whole sections of your project.
While playing around with a few different AIs, a question has arisen: What if I ask the same prompt? How different will the responses be between the different AIs? I’m looking to figure that out with a little experiment.
To limit the scope of this experiment, I have added a few constraints. First, I will only be using three different AIs. There are way too many options out there, and honestly, there is not enough time to test them all. Second, I will only be doing one prompt. I want to ask AI to generate code based on a semi-detailed description of what is needed. This way, I can let the AIs assume a couple of things, and in turn, that should create a little diversity.
What AIs Should We Use?
I want to choose a couple that are well-known, free to use, or at least have a free trial. I wanted the experience to feel realistic and practical. I think most people will do a quick web search for different AIs. They will choose a couple that look to meet their needs and then test drive before they spend any money. Given that, I have narrowed it down to the following:
1) ChatGPT 4.0
ChatGPT is probably the first AI most people were introduced to and the most known. Currently, it is free for version 4.0. I wanted to include this to give us a good baseline.
2) GitHub Copilot
Whenever I search for “Best AIs for code generation,” this one is at the top of the list. This option is a bit different than the others. It is an extension of your IDE and delivers more features than just a chatbot. It also has a paid monthly subscription but there is a 30-day trial. You can change the AI engine used, but for this experiment, I will keep it on the “GPT 4.0” option. Meaning the models for ChatGPT and GitHub Copilot will be the same for this experiment. I’m curious to see if the two will produce widely different results.
3) Claude.ai
This one is a bit of a pet choice. It has grown in popularity, and I have seen people say it is better for code generation than ChatGPT. Let’s see if that is true. Currently, version 3.5 is free to use.
What About the Prompt?
I will use a prompt that gives just enough detail without going off the deep end. I don’t want the prompt to contain all the methods we may need, or the parameters, and so on. I just want the AI to know that an API is needed; one that will be written in JavaScript for a given framework and follows the RESTful principles. I will not be asking for null checking or error handling. I want to see if the AI will suggest these things.
The following is the prompt I will be using:
Write me an API with endpoints for a grocery store’s inventory system. I need this written in JavaScript with an Express.js framework. It needs to be RESTful. The object has the following properties: id, name, category, quantity, and price.
I’m hopeful this will just be enough to achieve the desired result but let the outcome be different for the sake of comparing them.
Time to Test
I didn’t want any cached previous prompts to influence the results. To prevent this, I will only sign in if it is necessary and I will also remove any history of prior prompts where possible, either by deleting the prompt history on the AI’s site or using an incognito browser.
Results for ChatGPT:
1. const express = require('express'); 2. const app = express(); 3. const port = 3000; 4. 5. // Middleware to parse JSON bodies 6. app.use(express.json()); 7. 8. // Sample data to represent the inventory 9. let inventory = [ 10. { id: 1, name: 'Apple', category: 'Fruit', quantity: 100, price: 1.2 }, 11. { id: 2, name: 'Banana', category: 'Fruit', quantity: 150, price: 0.8 }, 12. { id: 3, name: 'Carrot', category: 'Vegetable', quantity: 200, price: 1.0 }, 13. ]; 14. 15. // Routes 16. 17. // 1. Get all inventory items 18. app.get('/inventory', (req, res) => { 19. res.json(inventory); 20. }); 21. 22. // 2. Get a specific inventory item by ID 23. app.get('/inventory/:id', (req, res) => { 24. const item = inventory.find(i => i.id === parseInt(req.params.id)); 25. if (!item) return res.status(404).send('Item not found'); 26. res.json(item); 27. }); 28. 29. // 3. Add a new inventory item 30. app.post('/inventory', (req, res) => { 31. const { name, category, quantity, price } = req.body; 32. 33. if (!name || !category || !quantity || !price) { 34. return res.status(400).send('Missing required fields'); 35. } 36. 37. const newItem = { 38. id: inventory.length + 1, 39. name, 40. category, 41. quantity, 42. price, 43. }; 44. 45. inventory.push(newItem); 46. res.status(201).json(newItem); 47. }); 48. 49. // 4. Update an existing inventory item by ID 50. app.put('/inventory/:id', (req, res) => { 51. const item = inventory.find(i => i.id === parseInt(req.params.id)); 52. if (!item) return res.status(404).send('Item not found'); 53. 54. const { name, category, quantity, price } = req.body; 55. if (name) item.name = name; 56. if (category) item.category = category; 57. if (quantity !== undefined) item.quantity = quantity; 58. if (price !== undefined) item.price = price; 59. 60. res.json(item); 61. }); 62. 63. // 5. Delete an inventory item by ID 64. app.delete('/inventory/:id', (req, res) => { 65. const index = inventory.findIndex(i => i.id === parseInt(req.params.id)); 66. if (index === -1) return res.status(404).send('Item not found'); 67. 68. const deletedItem = inventory.splice(index, 1); 69. res.json(deletedItem[0]); 70. }); 71. 72. // Start the server 73. app.listen(port, () => { 74. console.log(`Inventory API running on http://localhost:${port}`); 75. }); 76.
Beyond the code, ChatGPT provides some steps to follow. They include instructions on how to install Node.js and Express.js, the creation of the basic project, and how to test the endpoints with sample API calls. This is very helpful for someone new to development or these technologies. Having these steps will help me feel more comfortable trying to duplicate ChatGPT’s result.
The code returned has all the basic endpoints I was looking for: get all, get one, add new, update one, and delete one. The code itself is clean and readable. It also includes null checks on the methods that need them.
One thing that is missing is a patch method to handle a partial update on an item. Currently, there is only one put. This will do an update but isn’t the most efficient way if only a couple of values of the item are being changed.
There is also a case for the IDs to be a GUID instead of an integer or letting something else handle incrementation. The ID in the code is based on the length of the list of objects and adding one to it. This could potentially create a race condition if two people were adding a new item at the same time. For production code, I think most people will change the in-memory storage to some kind of database and let the database handle the auto-incrementation. It is my recommendation to do that or make it a GUID. Of course, this depends on your needs.
Results for GitHub Copilot:
1. const express = require('express'); 2. const app = express(); 3. const port = 3000; 4. 5. app.use(express.json()); 6. 7. let inventory = [ 8. { id: 1, name: 'Apple', category: 'Fruit', quantity: 100, price: 0.5 }, 9. { id: 2, name: 'Milk', category: 'Dairy', quantity: 50, price: 1.2 }, 10. ]; 11. 12. // Get all items 13. app.get('/inventory', (req, res) => { 14. res.json(inventory); 15. }); 16. 17. // Get a single item by ID 18. app.get('/inventory/:id', (req, res) => { 19. const item = inventory.find(i => i.id === parseInt(req.params.id)); 20. if (!item) return res.status(404).send('Item not found'); 21. res.json(item); 22. }); 23. 24. // Add a new item 25. app.post('/inventory', (req, res) => { 26. const newItem = { 27. id: inventory.length + 1, 28. name: req.body.name, 29. category: req.body.category, 30. quantity: req.body.quantity, 31. price: req.body.price, 32. }; 33. inventory.push(newItem); 34. res.status(201).json(newItem); 35. }); 36. 37. // Update an existing item 38. app.put('/inventory/:id', (req, res) => { 39. const item = inventory.find(i => i.id === parseInt(req.params.id)); 40. if (!item) return res.status(404).send('Item not found'); 41. 42. item.name = req.body.name; 43. item.category = req.body.category; 44. item.quantity = req.body.quantity; 45. item.price = req.body.price; 46. 47. res.json(item); 48. }); 49. 50. // Delete an item 51. app.delete('/inventory/:id', (req, res) => { 52. const itemIndex = inventory.findIndex(i => i.id === parseInt(req.params.id)); 53. if (itemIndex === -1) return res.status(404).send('Item not found'); 54. 55. const deletedItem = inventory.splice(itemIndex, 1); 56. res.json(deletedItem); 57. }); 58. 59. app.listen(port, () => { 60. console.log(`Inventory app listening at http://localhost:${port}`); 61. }); 62.
We have a very similar result for GitHub Copilot. All the same methods are here And the error handling looks the same. However, there was no null checking from the Copilot code. This can lead to unhandled errors and confusion for the end users. The other noticeable difference is that ChatGPT destructured its objects and GitHub Copilot didn’t. This could be seen as a personal preference, but destructuring is easier to read and helps avoid repetitive access. The same issue with the ID is present as well. There is also still no patch method for handling a partial update of an item.
As for the additional tasks beyond the code, Copilot gives me very little compared to ChatGPT. It provides a four-bullet-point list of how to run the code:
This might be fine for someone with prior experience but doesn’t provide all the details a new developer needs. I would like to see more about installing Node.js and testing the endpoints with some examples. Copilot is marketed more toward developers, and they seem to assume the end users have some level of technical knowledge.
Claude.ai Result:
1. const express = require('express'); 2. const bodyParser = require('body-parser'); 3. const { v4: uuidv4 } = require('uuid'); 4. 5. const app = express(); 6. const PORT = process.env.PORT || 3000; 7. 8. // Middleware 9. app.use(bodyParser.json()); 10. 11. // In-memory storage for groceries (replace with a database in production) 12. let groceryInventory = [ 13. { 14. id: uuidv4(), 15. name: 'Apples', 16. category: 'Produce', 17. quantity: 100, 18. price: 0.50 19. }, 20. { 21. id: uuidv4(), 22. name: 'Milk', 23. category: 'Dairy', 24. quantity: 50, 25. price: 3.99 26. } 27. ]; 28. 29. // GET all grocery items 30. app.get('/api/groceries', (req, res) => { 31. res.json(groceryInventory); 32. }); 33. 34. // GET a specific grocery item by ID 35. app.get('/api/groceries/:id', (req, res) => { 36. const item = groceryInventory.find(g => g.id === req.params.id); 37. 38. if (!item) { 39. return res.status(404).json({ message: 'Grocery item not found' }); 40. } 41. 42. res.json(item); 43. }); 44. 45. // CREATE a new grocery item 46. app.post('/api/groceries', (req, res) => { 47. const { name, category, quantity, price } = req.body; 48. 49. // Validate input 50. if (!name || !category || quantity === undefined || price === undefined) { 51. return res.status(400).json({ message: 'Missing required fields' }); 52. } 53. 54. const newItem = { 55. id: uuidv4(), 56. name, 57. category, 58. quantity, 59. price 60. }; 61. 62. groceryInventory.push(newItem); 63. res.status(201).json(newItem); 64. }); 65. 66. // UPDATE an existing grocery item 67. app.put('/api/groceries/:id', (req, res) => { 68. const { id } = req.params; 69. const { name, category, quantity, price } = req.body; 70. 71. const itemIndex = groceryInventory.findIndex(g => g.id === id); 72. 73. if (itemIndex === -1) { 74. return res.status(404).json({ message: 'Grocery item not found' }); 75. } 76. 77. // Validate input 78. if (!name || !category || quantity === undefined || price === undefined) { 79. return res.status(400).json({ message: 'Missing required fields' }); 80. } 81. 82. groceryInventory[itemIndex] = { 83. id, 84. name, 85. category, 86. quantity, 87. price 88. }; 89. 90. res.json(groceryInventory[itemIndex]); 91. }); 92. 93. // PARTIALLY UPDATE an existing grocery item 94. app.patch('/api/groceries/:id', (req, res) => { 95. const { id } = req.params; 96. 97. const itemIndex = groceryInventory.findIndex(g => g.id === id); 98. 99. if (itemIndex === -1) { 100. return res.status(404).json({ message: 'Grocery item not found' }); 101. } 102. 103. // Merge existing item with update 104. groceryInventory[itemIndex] = { 105. ...groceryInventory[itemIndex], 106. ...req.body 107. }; 108. 109. res.json(groceryInventory[itemIndex]); 110. }); 111. 112. // DELETE a grocery item 113. app.delete('/api/groceries/:id', (req, res) => { 114. const { id } = req.params; 115. 116. const initialLength = groceryInventory.length; 117. groceryInventory = groceryInventory.filter(g => g.id !== id); 118. 119. if (groceryInventory.length === initialLength) { 120. return res.status(404).json({ message: 'Grocery item not found' }); 121. } 122. 123. res.status(204).send(); 124. }); 125. 126. // Start the server 127. app.listen(PORT, () => { 128. console.log(`Server running on port ${PORT}`); 129. }); 130. 131. // Error handling middleware 132. app.use((err, req, res, next) => { 133. console.error(err.stack); 134. res.status(500).json({ 135. message: 'Something went wrong!', 136. error: err.message 137. }); 138. }); 139. 140. module.exports = app; 141.
This time, the code has all the same methods but includes a method for performing a partial update to an item. This is nice to see and something I wanted from the other two AI. Patch doesn’t replace the item; it only modifies the parts of the item that need to be updated. This is the preferred way to handle partial updates.
The code includes null checks where needed as well as error handling. Claude handles the ID field differently than the other two AIs, choosing to use a GUID to represent the ID’s value.
Another difference is the use of “filter” to handle the delete instead of “findIndex” and “splice.” This approach can be a bit harder to read initially and requires checking the length of the new array versus the initial array for verification. I don’t see the benefit here and prefer using “findIndex” to first determine if the item is present in the array before performing the delete. This allows us to throw an error before the delete action happens if the item is not present in the array. The “filter” in JavaScript also creates a new copy of the array where “splice” modifies the original. If your array is larger in size it could lead to performance and memory issues. Additionally, with “findIndex” and “splice,” we can return the deleted item, which might be useful if you need to present the user with information about the item that was removed.
Conclusion
All in all, the AIs provide me with a working solution. There are parts I like about each. If I were to give a nod to one, it would be the result from Claude.ai. This is because it includes null checks and implements a patch method.
I recommend not relying on just one AI for tasks like this. One AI might miss parts that another includes. You should pick and choose the parts you like from each to create a solution that works for your needs. You might even come across techniques you haven’t seen before and learn something new in the process.
Another thing this exercise highlights is the importance of continued prompting after the initial request. For example, after providing the initial prompt, you could follow up and ask the AI to add null checks, use GUIDs for IDs, or even integrate database connectivity. There may be times the code doesn’t match your local coding standards or you may need to add more detail to your requests to refine the code to meet all your needs.
With the use of AI, we need to be cautious and consider the potential ramifications of relying on it too much. As I’ve pointed out a couple of times, continued prompting with more details can help refine the results. This can only be done effectively if the person providing the prompts understands the code being generated. They will need to know what’s missing or unnecessary. AI can be a useful tool, but it must be used correctly. As developers, we need to treat it as a tool for continued learning, using it with critical thinking rather than simply relying on it.

Michael Erskin
Michael Erskin is a seasoned software engineer with approximately 10 years of experience. He specializes in developing websites and mobile applications using JavaScript. In his free time, he enjoys keeping up with the latest tech, gaming, and spending quality time with his two daughters. For inquiries, feel free to contact him at merskin@inspyrsolutions.com.
Share This Article
Contact Us
We’re here for you when you need us. How can we help you today?