bj/anki-csv2ankicards
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

overhauled the project to get away from files (a little)

Browse Source
This commit is contained in:
B.J. Dweck 2023-09-11 19:09:18 +03:00
parent a13f92548c
commit c3bdf2dd5e
11 changed files with 247 additions and 243 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -2,3 +2,4 @@ venv/
*.pyc
__pycache__/
.env

27
.vscode/launch.json vendored Normal file
View File

@ -0,0 +1,27 @@
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Python: Flask",
"type": "python",
"request": "launch",
"module": "flask",
"envFile": "${workspaceFolder}/.env",
"env": {
"FLASK_APP": "server.py",
"FLASK_ENV": "development",
"FLASK_DEBUG": "0"
},
"args": [
"run",
"--no-debugger",
"--no-reload"
],
"jinja": true,
"justMyCode": true
}
]
}

120
README.md
View File

@ -1,26 +1,24 @@
# csv2ankicards
# AnkiAI
A comprehensive toolkit that offers:
- Conversion of CSV files into Anki deck packages (.apkg files).
- Conversion of image files in a directory to a text file using Optical Character Recognition (OCR).
- Generation of CSV format question-answer pairs from textual content using OpenAI's GPT-3 model.
- **RESTful API endpoint to upload and convert multiple images directly into an Anki deck package.**
AnkiAI is a robust system that converts images containing text into structured Anki cards using Optical Character Recognition (OCR) and OpenAI's GPT-4 language model. Users can quickly generate decks of flashcards from their images for effective study.
## Features
- Converts image content to textual content using OCR.
- Uses OpenAI's GPT-4 model to structure the content into Anki decks and cards.
- Outputs the structured content as an Anki package.
- Converts a CSV file with questions and answers into an Anki deck package.
- Converts image files from a specified directory to a single text file using OCR.
- Generates CSV formatted question-answer pairs based on a given text content, ideal for studying or summarization.
- For CSV: there are only two columns in the CSV file, separated by the first comma encountered.
- CSV files should have a "Front" column for questions and a "Back" column for answers.
- **API endpoint that accepts multiple image uploads, processes them through the pipeline, and returns an Anki deck package.**
## Dependencies
- genanki: Used for creating Anki decks and cards.
- Pillow: Image processing library.
- openai: API library for OpenAI's GPT-4 model.
- flask: Web server to host the service.
## Installation
## Setup and Installation
1. Clone this repository:
```bash
git clone https://git.rudefox.io/bj/anki-csv2ankicards.git
cd csv2ankicards
git clone https://git.rudefox.io/bj/anki-json2ankicards.git
cd json2ankicards
```
2. Set up a virtual environment and activate it:
@ -34,94 +32,36 @@ A comprehensive toolkit that offers:
pip install -r requirements.txt
```
## Configuration
Before using the `text2csvdeck.py` script, ensure that you have set the `OPENAI_API_KEY` environment variable:
4. Set up the OpenAI API key:
```bash
export OPENAI_API_KEY=your_openai_api_key_here
export OPENAI_API_KEY=your_openai_api_key
```
Remember to replace `your_openai_api_key_here` with your actual OpenAI API key.
## Usage
### **REST API Usage**
To start the server:
5. Run the server:
```bash
python server.py
```
#### Endpoint: `/deck-from-images`
## Usage
**Method**: POST
1. Start the server as mentioned above.
**Description**: Accepts multiple image uploads, processes them through the pipeline, and returns an Anki deck package.
2. Use a tool like [Postman](https://www.postman.com/) or `curl` to send images to `http://localhost:5000/deck-from-images` as a multi-part POST request.
**Body**:
3. The server will respond with a downloadable Anki package. Import this into your Anki app and start studying!
- form-data with key `image` and multiple image files as values.
## Modules
**Response**:
1. **ankiai.py**: The main module that orchestrates the flow.
2. **images2text.py**: Converts image content into text using OCR.
3. **json2deck.py**: Converts structured JSON data into an Anki package.
4. **prompt4cards.py**: Uses OpenAI to structure the content into Anki decks and cards.
5. **server.py**: Flask server to host the service.
- An `output.apkg` file ready for import into Anki.
## Contributing
### Pipeline Usage
Contributions are welcome! Please submit a pull request or open an issue to discuss changes or fixes.
To convert a directory of images directly to an Anki deck package:
## License
```bash
python pipeline.py /path/to/your/image_directory/
```
This will process the images, extract text, convert text to a set of questions and answers in CSV format, and then produce an `output.apkg` file ready for import into Anki.
### Image to Text Conversion
To convert images from a directory to a single text file using OCR:
```bash
python images2text.py /path/to/your/image_directory/
```
This will produce a `final.txt` file which contains the text extracted from the images.
#### Supported Image Formats
Currently supported formats for the images are: `.png`, `.jpg`, and `.jpeg`.
### Text to CSV Deck Generation
To generate a CSV deck of question-answer pairs from a given text file:
```bash
python text2csvdeck.py /path/to/your/textfile.txt
```
This will analyze the content of the given text file and generate a corresponding `_deck.csv` file with questions and answers that capture the main points and themes of the text.
**Note:** This script uses the OpenAI GPT-3 model. Ensure you have the necessary API key and OpenAI Python client installed.
### CSV to Anki Conversion
To convert a CSV file into an Anki deck package:
```bash
python csv2ankicards.py /path/to/your/csvfile.csv output.apkg
```
This will produce an `output.apkg` file which can then be imported into Anki.
#### CSV Format
The CSV file should follow this format:
```
Front,Back
Your question here,Your answer here
Another question,list of: answer1, answer2, answer3
...
```
**Note:** If your answers contain commas, they will be considered as part of the answer. Only the first comma is used to separate the question from the answer.
[MIT License](LICENSE)

21
ankiai.py Normal file
View File

@ -0,0 +1,21 @@
import sys
from images2text import main as ocr_images
from prompt4cards import prompt_for_card_content, response_to_json
from json2deck import to_package
def images_to_package(directory_path, outfile):
ocr_text = ocr_images(directory_path)
response_text = prompt_for_card_content(ocr_text)
deck_json = response_to_json(response_text)
to_package(deck_json).write_to_file(outfile)
print(f"Deck created at: {outfile}")
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python ankiai.py <directory_path_containing_images>")
sys.exit(1)
images_to_package(sys.argv[1])

49
csv2ankicards.py
View File

@ -1,49 +0,0 @@
import csv
import genanki
import sys
# Create a new model for our cards. This is necessary for genanki.
MY_MODEL = genanki.Model(
1607392319,
"Simple Model",
fields=[
{"name": "Question"},
{"name": "Answer"},
],
templates=[
{
"name": "Card 1",
"qfmt": "{{Question}}",
"afmt": "{{FrontSide}}<hr id='answer'>{{Answer}}",
},
])
def csv_to_anki(csv_path, output_path):
with open(csv_path, 'r', encoding='utf-8') as f:
reader = csv.reader(f)
# Skipping the header row
next(reader, None)
my_deck = genanki.Deck(2059400110, "CSV Deck")
for row in reader:
# Use row directly without splitting
question = row[0]
answer = ",".join(row[1:])
note = genanki.Note(
model=MY_MODEL,
fields=[question, answer]
)
my_deck.add_note(note)
genanki.Package(my_deck).write_to_file(output_path)
if __name__ == "__main__":
if len(sys.argv) != 3:
print("Usage: python convert.py <input_csv> <output_apkg>")
sys.exit(1)
input_csv = sys.argv[1]
output_apkg = sys.argv[2]
csv_to_anki(input_csv, output_apkg)
print(f"Deck created at: {output_apkg}")

2
images2text.py
View File

@ -80,7 +80,7 @@ def main(directory_path):
f.write("\n".join(final_text))
print(f"All images processed! Final output saved to {FINAL_OUTPUT}")
return FINAL_OUTPUT # Add this line
return final_text # Add this line
if __name__ == "__main__":

55
json2deck.py Normal file
View File

@ -0,0 +1,55 @@
import json
import genanki
import sys
# Create a new model for our cards. This is necessary for genanki.
MY_MODEL = genanki.Model(
1607372319,
"Simple Model",
fields=[
{"name": "Title"},
{"name": "Question"},
{"name": "Answer"},
],
templates=[
{
"name": "{{Title}}",
"qfmt": "{{Question}}",
"afmt": "{{FrontSide}}<hr id='answer'>{{Answer}}",
},
])
def json_file_to_package(json_path):
with open(json_path, 'r', encoding='utf-8') as f:
json_data = json.load(f)
package = to_package(json_data)
return package
def to_package(deck_json):
deck_title = deck_json["DeckTitle"]
deck = genanki.Deck(1607372319, deck_title)
for card_json in deck_json["Cards"]:
title = card_json["Title"]
question = card_json["Question"]
answer = card_json["Answer"]
note = genanki.Note(
model=MY_MODEL,
fields=[title, question, answer]
)
deck.add_note(note)
return genanki.Package(deck)
if __name__ == "__main__":
if len(sys.argv) != 3:
print("Usage: python convert.py <input_json> <output_apkg>")
sys.exit(1)
input_json = sys.argv[1]
output_apkg = sys.argv[2]
json_file_to_package(input_json).write_to_file(output_apkg)
print(f"Deck created at: {output_apkg}")

27
pipeline.py
View File

@ -1,27 +0,0 @@
import sys
import os
from images2text import main as images_to_text
from text2csvdeck import text_file_to_csv_deck
CSV_DECK_NAME = "output_deck.csv"
APKG_NAME = "output.apkg"
def pipeline(directory_path):
# 1. Convert images in the directory to a text file
text_file_name = images_to_text(directory_path)
# 2. Convert the text file to a CSV deck using ChatGPT
text_file_to_csv_deck(text_file_name)
# 3. Convert the CSV deck to an Anki package
os.system(f"python csv2ankicards.py {CSV_DECK_NAME} {APKG_NAME}")
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python pipeline.py <directory_path_containing_images>")
sys.exit(1)
pipeline(sys.argv[1])

104
prompt4cards.py Normal file
View File

@ -0,0 +1,104 @@
import openai
import sys
import os
import json
CHAT_MODEL = "gpt-3.5-turbo"
OUTPUT_FILENAME = "output_deck.json"
API_KEY = os.environ.get("OPENAI_API_KEY")
if not API_KEY:
raise ValueError("Please set the OPENAI_API_KEY environment variable.")
openai.api_key = API_KEY
# Given prompt template
PROMPT_TEMPLATE = """
Please come up with a title for the deck and a set of 10 index cards for memorization,
including a title, front, and back for each card. The index cards should completely
capture the main points and themes of the text. In addition, they should contain any
numbers or data that humans might find difficult to remember. The goal of the index
card set is that one who memorizes it can provide a summary of the text to someone
else, conveying the main points and themes.
You will provide the deck title, and the titles, questions, and answers for each card
in a structured format as follows:
```
Deck Title: Title of the Deck
Cards:
- Title: Card Title 1
Front: What is the capital of New York?
Back: Albany
- Title: Card Title 2
Front: Where in the world is Carmen San Diego?
Back: Nobody knows
```
{content}
"""
def prompt_for_card_content(text_content):
# Prepare the prompt
prompt = PROMPT_TEMPLATE.format(content=text_content)
# Get completion from the OpenAI ChatGPT API
response = openai.ChatCompletion.create(
model=CHAT_MODEL,
messages=[
{"role": "user", "content": prompt}
],
temperature=0,
)
# Extract content from response and save to a new file
return response.choices[0]['message']['content']
def response_to_json(response_text):
lines = [line.strip() for line in response_text.split("\n") if line.strip()]
deck_title = None
cards = []
current_card = {}
for line in lines:
if "Deck Title:" in line and not deck_title:
deck_title = line.split("Deck Title:", 1)[1].strip()
elif "Title:" in line:
if current_card: # If there's a card being processed, add it to cards
cards.append(current_card)
current_card = {}
current_card["Title"] = line.split("Title:", 1)[1].strip()
elif "Front:" in line:
current_card["Question"] = line.split("Front:", 1)[1].strip()
elif "Back:" in line:
current_card["Answer"] = line.split("Back:", 1)[1].strip()
if current_card: # Add the last card if it exists
cards.append(current_card)
return {
"DeckTitle": deck_title,
"Cards": cards
}
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python text2jsondeck.py <text_file_path>")
sys.exit(1)
text_file_path = sys.argv[1]
# Read the text content
with open(text_file_path, 'r') as file:
text_content = file.read()
response_text = prompt_for_card_content(text_content)
deck_json = response_to_json(response_text)
with open(OUTPUT_FILENAME, 'w') as json_file:
json.dump(deck_json, json_file)
print(f"Saved generated deck to {OUTPUT_FILENAME}")

6
server.py
View File

@ -4,7 +4,7 @@ import os
import tempfile
import shutil
from pipeline import pipeline
from ankiai import images_to_package
app = Flask(__name__)
@ -16,9 +16,11 @@ def save_uploaded_images(images, directory):
for img in images:
# Sanitize the filename
safe_filename = secure_filename(img.filename)
if not safe_filename:
# Handle the case where the filename becomes empty after sanitization
raise ValueError("Invalid filename")
filename = os.path.join(directory, safe_filename)
img.save(filename)
@ -35,7 +37,7 @@ def deck_from_images():
save_uploaded_images(images, TEMP_DIR)
try:
pipeline(TEMP_DIR)
images_to_package(TEMP_DIR, OUTPUT_FILE)
return send_from_directory('.', OUTPUT_FILE, as_attachment=True)
except Exception as e: # Consider catching more specific exceptions
return jsonify({'error': str(e)}), 500

70
text2csvdeck.py
View File

@ -1,70 +0,0 @@
import openai
import sys
import os
CHAT_MODEL = "gpt-3.5-turbo"
OUTPUT_FILENAME = "output_deck.csv"
API_KEY = os.environ.get("OPENAI_API_KEY")
if not API_KEY:
raise ValueError("Please set the OPENAI_API_KEY environment variable.")
openai.api_key = API_KEY
# Given prompt template
PROMPT_TEMPLATE = """
Please come up with a set of 10 index cards for memorization, including front and back.
The index cards should completely capture the main points and themes of the text.
In addition, they should contain any numbers or data that humans might find difficult to remember.
The goal of the index card set is that one who memorizes it can provide a summary of the text to someone else, conveying the main points and themes.
You will provide the questions and answers to me in CSV format, as follows:
```
Front,Back
What is the capital of New York?,Albany
Where in the world is Carmen San Diego?,Nobody knows
```
The question/answer pairs shall not be numbered or contain any signs of being ordered.
{content}
"""
def text_file_to_csv_deck(text_file_path):
# Read the text content
with open(text_file_path, 'r') as file:
text_content = file.read()
content_to_csv(text_content)
def content_to_csv(text_content):
# Prepare the prompt
prompt = PROMPT_TEMPLATE.format(content=text_content)
# Get completion from the OpenAI ChatGPT API
response = openai.ChatCompletion.create(
model=CHAT_MODEL,
messages=[
{"role": "user", "content": prompt}
],
temperature=0,
)
# Extract CSV content from response and save to a new file
csv_content = response.choices[0]['message']['content']
with open(OUTPUT_FILENAME, 'w') as csv_file:
csv_file.write(csv_content)
print(f"Saved generated deck to {OUTPUT_FILENAME}")
if __name__ == "__main__":
if len(sys.argv) != 2:
print("Usage: python text2csvdeck.py <text_file_path>")
sys.exit(1)
text_file_to_csv_deck(sys.argv[1])