[IMPORTANT UPDATE] Come help the grading system! Overhauled with Benchmark Books, Movies & TV Shows

Korean
1

[UPDATE - See below]

Hi all,

So, I’ve been thinking about recalculating the Korean difficulty ratings from a new notion of ‘benchmarks’ which I’ve developed from my experiences opening other languages. I think it’ll really improve the accuracy of the system, but I need a little help from you all!

Next Step:

We need to choose a series of books, movies & TV shows which will serve as the ‘benchmark’ for the grading system. As an example, I always choose Harry Potter volume 1 and put it at lvl 29/30 (~TOPIK 5). This makes it similar to the Japanese Grading system.

Once we have a series of book set at certain levels, then the grading system starts to take shape.

So any suggestions? What books & movies & tv shows would be appropriate for a particular TOPIK level? Please let me know below!

[UPDATE]

So per your suggestions, i’ve set bookmarks as shown below and recalculated the entire rating system. Please take a look and see what you think about how the gradings landed.
I know, this will be a little chaotic but I think the end result will be much better!

Please let me know if you have more suggestions for benchmarks, it’ll really help us out!

Significant changes:

  • You will see that Harry Potter, 소리를 보는 소년, among other things have been bumped up, per your suggestions
  • Many of the TV benchmarks are simply coming from the Japanese side - I took that level and assumed the Korean level would be relatively close
  • You will notice that you may have a lot more gradings all of a sudden. If you help us set benchmarks, those will reduce in volume, so please help us out :slight_smile:

Benchmark Books

Benchmark TV Shows

Benchmark Movies

FWIW I know TOPIK doesn’t correspond perfectly to native material, but just think of it as a general recommendation to someone at a particular TOPIK level

3 Likes
2

Ah, I didn’t realize it was a graded reader. And now I don’t know how B2 corresponds to TOPIK. Probably TOPIK5? :sweat_smile:

2 Likes
3

That’s what I meant. The thing that’s weird to me is that it gets compared to pretty much everything I read. Just from my most recent gradings:

  • L19, L17, that makes sense ;
  • L7, L14, L12, L10, that’s what I don’t get.

(All of these already had stable gradings btw, apart maybe from the L7 one)

Kind of, yeah. The difficulty would be exactly where to put it in B2. The first three of the series have been at the top of that level, but for this one I’m not sure. If we say that 소리를 보는 소년 is level 25, then 춘향전 is definitely below that. But then maybe 소리를 보는 소년 should be set higher, I’m not sure. It’s the same for 전우치전, it is definitely easier than 소리를 보는 소년, and I’d say around the level of 춘향전 I guess. Kinda hard to compare since one has hard vocab and relatively easier grammar, and the other one is the opposite.

2 Likes
4

FYI @HopeWaterfall I moved this comment in this thread, so the discussion continues here :slight_smile:

2 Likes
5

I’d guess:

A1 - TOPIK 1
A2 - TOPIK 2
B1 - TOPIK 3
B2 - TOPIK 4
C1 - TOPIK 5
C2 - TOPIK 6

2 Likes
6

I think there’s more of a direct comparison between TOPIK lvl and CEFR lvl. At least this paper seems to say that.

2 Likes
7

Books:

무민 시리즈 (series) | L10
호랑이 들어와요 (series) | L20
이상한 과자 가게 전천당 (series) | L20??
또다시 같은 꿈을 꾸었어 | L24
책벌레의 하극상 (series) | L29
어린왕자 (공식 한국어판) | L25??

You could perhaps use the Japanese levels for translated books since they’ll be more accurate as more users have graded them.

TV Shows:

페파 피그 | L10?? (L5)
브레드 이발소 | L15 (L10-12)
미라큘러스 레이디버그 | L18

3 Likes
8

Just weighing in here as someone in the unique position of being very familiar with the Japanese grading system, having learned a European language and taken language tests in it and thus being very familiar with the CEFR system, and having worked my way through Korean learning materials up through TOPIK 6…

I would be tempted to bump the TOPIK to CEFR correspondence levels down ever so slightly, as follows:

TOPIK 1: A1
TOPIK 2: A2
TOPIK 3: B1
TOPIK 4: B1.5
TOPIK 5: B2
TOPIK 6: B2.5 ~ C1

My reasoning for that is the scope of grammar covered by the complete TOPIK spectrum does not ever get into very advanced or formal constructions that you can find in a textbook like KGIU advanced, and those constructions are usually a big focus of the CEFR C levels. I feel that the scope of vocab for the TOPIK is also narrower than it is for CEFR C levels. That is just my opinion though which doesn’t matter much in the grand scheme of things… it would probably be easier to keep the TOPIK levels as is corresponding nearly to the CEFR. :sweat_smile:

In terms of actual benchmarks, though — I’ve read a number of translated books in Korean that I previously read in Japanese and found the levels between them quite similar, so that may be a good place to start? Mileage may vary though, as I am obviously biased with regard to understanding better as I’ve read the story before and am very comfortable deciphering 한자어 from my Japanese background. But, I guess we don’t need to get so scientific starting out here!

Some titles to check out that could be used as benchmarks:

야시 (夜市)
편의점 인간 (コンビニ人間)
고백 (告白)
보라색 치마를 입은 여자 (紫のスカートの女)

In addition, there’s some discussion within this thread about the level of 소리를 보는 소년. I finished it and graded it and it ended up at level 25 which felt just a bit low. Considering the grammar and vocab I am tempted to say that it felt almost like TOPIK 5, because there were some confusing honorifics and also “difficult” historical vocab and idioms…maybe level 28 or something feels more accurate? :thinking: That could just mean I did a bad job grading it though…

Sorry if this response as not at all helpful, I realize it’s a lot of my opinions and musings. :laughing:

6 Likes
9

Commenting to add further, this may be a controversial hot take (or maybe not?), but I would say TOPIK 6 is almost roughly equivalent to JLPT N1, and on the Japanese side we have N1+ as a category when adding books, so something like that may be an idea! I don’t think the Korean side of the site has as many advanced readers, though…and that’s just the opinion of one person. :laughing: So I’m seriously happy with whatever!

4 Likes
10

FWIW, I was also tempted to bump it down one level, just like I did for Japanese JLPT. JLPT, as i’m sure you know, also claims to be equivalent to CEFR (JLPT N1 = C2), which I think everyone generally agrees is stretching it.

All that really matters here is what do we want the level mapping to TOPIK to be and that the levels are relatively similar across languages. I think the current levels are fine, as long as we think equivalent things are in the same ballpark (is Harry Potter at lvl 30 (TOPIK5) a good recommendation?).

Also, really appreciate the perspective @bungakushoujo… especially since it coincides with my suspicion/feeling too :joy:

4 Likes
11

Fist bump for that :facepunch:t2:

I’d be pretty happy with that as it’d mean consistency + the TOPIK levels as a guideline for picking books.

That sounds pretty reasonable to me. That would make my assessment of 소리를 보는 소년 at level 27/28 ish have it come out at between TOPIK 4-5, which feels accurate. :slightly_smiling_face:

3 Likes
12

I’ve added an update to the original post - please read :slightly_smiling_face:. The grading system reprocessing will still take a little while (maybe 30 minutes?) so things may be a little weird looking until then.

I will edit this comment when it’s done.

A few things I’ll call out:

Let me know what you generally think :slight_smile:

Edit: The processing is mostly done. We are still processing search results (you won’t see everything in search until an hour or two passes…), but you should be able to generally explore :slight_smile:

3 Likes
13

That’s more a problem with the book than with our gradings. The short stories for these intermediate Korean short stories are the same as the ones he uses for his beginner short stories in other languages. That’s why Korean doesn’t have any beginner short stories in Korean by Olly Richards. Maybe L9 is a little low considering vocab, but for grammar it doesn’t have a single intermediate grammar point and all sentences are pretty short and straightforward. Plus, a lot of vocab is defined, which makes reading it a lot easier.

I’m guessing that it’s called intermediate instead of beginner because if you’re learning for example Spanish from English, you’ll be able to read a lot more easily than learning Korean from English.

(I feel like the Spanish level is a bit high currently, based on what I’ve heard about it, and the difficulty of the Italian one to me. Although I guess I’m not the best judge of that with my prior knowledge in Romance languages making reading a lot easier.)

I just started rewatching 신데렐라와 네명의 기사 | L30??, I feel like it’d be a good TOPIK 4 benchmark. Vocab is pretty simple, and mostly related to daily life. For grammar, there are various politeness levels used, but nothing too crazy either. There’s also technically one character that speaks in dialect, but it’s a side character that doesn’t appear often, so I don’t feel like that would bump the overall level.

1 Like
14

Ah, that’s interesting. Perhaps I’ll bump it up a little then to lvl 12 or lvl 13? :thinking:

@bibliothecary do you have any thoughts?

Now that I look at it again, it was the beginner stories that were quoted as A2/B1, but perhaps that’s too high? That spanish rating is purely the benchmark rating right now, so could definitely move it down. Would lvl 13 make sense (early A2?).

Excellent. I’ll put that as lvl 24 then?

In other news, I realized some of those auto-generated benchmarks from the Japanese rating were accidentally plucked from the Korean side before I wiped the ratings, not the Japanese side, oops. Will need to rerun this calculation again.

1 Like
15

I agree with @HopeWaterfall, it’s A2 at most, but I don’t see the harm in bumping up a couple of levels if need be.

2 Likes
16

Yeah, I feel like that would make sense.

Probably? This is kinda hard for me to evaluate, because my Spanish level is too high compared to this. Like I said though, if I’m just looking at the Italian version, I can read it with a little bit of guesswork. And as much as kowing French, Spanish and some Latin helps, I feel like calling my not even 2 months of Italian B1 would be outrageous.

Yeah, seems right!

2 Likes
17

Is the grading system still recalculating? My finished books have all returned to temporary ratings:

download (1)
download (1)687×522 42.4 KB

I’m also being asked to grade https://learnnatively.com/book/63d9687978/ against an ever-growing number of books - I had already graded it as higher than other books I had read in the L20s, but now it’s comparing with others starting at L0 (this isn’t extra grading, I clicked on the blue button on the dashboard):

download
download600×690 57 KB

18

Unfortunately no. This is the recalculation. TBH, we just need more benchmarks… Korean readership is way more spread out among a lot of books, which means more benchmarks will be necessary.

Since we do have a lot of gradings, I’m in the process of trying to figure out some potential benchmarks based on the data we have.

If, for example, we go through the 두루책방 series and simply set a bunch of books at a certain level, i’m guessing that will have a large impact.

Edit I know the grading is a pain right now… but hopefully with the additional tools and approach we’re building here, it’ll be a big help to the Korean grading system and future grading systems :slightly_smiling_face:

2 Likes
19

regraded what it gave me but it wasnt much tbh. I have no concept of the topik levels or what’s considered hard/easy grammar tbh so I can’t really suggest any benchmarks :sweat_smile:

2 Likes
20

Is it still in recalculation? A lot of levels that were set are now uncertain, such as the 두루책방 levels. I just graded 최고의 이름 | L12?? with 101 auto-generated gradings, but this book is now both graded as a similar level to L2 books and L24?? books.